High Profile MSP Backup Failures

Published 6 days ago5 min readKey Drivers Behind MSPs Transitioning to Halo PSA HERE...

High profile backup failures will be today's hot topic. By collecting a number of horror stories relating to instances where organizations have utterly failed in their ability to recover from a backup failure, it is my aim to demonstrate that we are no longer out here playing games.

It is time to take whatever backup strategy you are offering to clients and initiate a good going over, a complete top town backwards 360 evaluation if you will. It is my aim to put the fear of god into the readers of the article here and if it creates a situation where even one service provider decides to take action then it will have been worth it and I will achieve my life long goal of being considered a hero.

While you are here, Take a look at some of our other IT Computer Consulting related articles below that may interest you:

The goal of this article is twofold with this collection of nightmare fuel. First it is to facilitate action in the form of taking backup and recovery solutions seriously and to encourage a serious review of backup strategy. Secondly to encourage documentation of recovery procedures. It is one thing to undertake a client's backup but the professionals in our industry fully test and document the recovery procedures.

They also wrap both the backup and recovery solutions up in a little bow in the form of documentation as well as a good quality cyber insurance policy because no matter how great you are, there is some crusty old judge out there who is having their Friday golf game being encroached upon by some service provider thinking that they deserve a fair hearing.

United Airlines Backup Failure - 2015

It was a crisp yet cloud free morning, just like any other when United suffered what they termed a “connectivity issue” which affected thousands of passengers across the planet. The culprit? A faulty router went rogue and this caused a delay of 90 minutes which of course may not seem like a big deal however for a company the size of United meant millions of dollars lost in canceled flights.

Delta Airlines Backup Failure - 2017

Not to be outdone and thankfully no planes fell out of the sky on this one either, however a configuration problem which is a nice way of saying somebody pressed the wrong button on their computer mouse, caused a power outage at their main data center.

As any company would, they whipped out their computer backups and as quick as a flash, nothing happened. The company's disaster recovery plan performed as well as a plane with one wing and caused significant downtime. Someone did not set the backups correctly in the first place and because no test restores occurred. Total cost of this incident was in the order of $25 million. I wonder if the IT Support tech responsible was able to talk his way out of that one.

British Airways Backup Failure - 2018

This incident demonstrates why it is really worth investing in an effective cyber security insurance policy as it shows that even if it is not your fault, you can still be held responsible.

The data breach was due to a third party utility used on the website of the company which redirected visitors to another site that the scoundrels controlled. Approximately 380K customers were impacted over the month this breach occurred.

The criminals went on a no expenses spared shopping spree with the credit card information of the customers of BA before being detected.

British Airways received a fine in the order of about $350 million. The fine was based on 1.5% of the organization's worldwide turnover. One thing you have to admit here, the governments must be rubbing their hands together every time some sort of breach occurs. They make more money than the criminals.

There really does seem to be a kick them while their down type of behavior.

TSB Bank Backup Failure - 2018

This story should make any IT service provider lay awake at night especially if you have a server migration or network infrastructure upgrade scheduled.

They were doing a system migration to a new computer system when all hell broke loose. Every part of the banking system was impacted by the failed migration including phones and computer terminals in branch locations.

Guess how long it took to recover from this little indiscretion? 8 long months to recover to the point the computer network was back to where it was. The fines alone from regulatory organizations were in the order of $100 million large ones along with around $60 million in compensation for their customers.

Perhaps if the bank had outsourced their IT operations and the people undertaking the migration had a tested disaster recovery process then this expensive lesson could have been avoided.

Total cost of this oversight was in the order of $700 million.

Conclusion

This is but a small selection of high profile backup failures. Companies of the size listed here stand to lose millions of dollars per day if they do not have multiple points of failure and fully tested restoration plans. I also show that the biggest winner in any serious backup failure or data breach is the government. For them it is like seeing the Wonka golden ticket. To me that just does not seem right.

There has to be some way to ensure companies adhere to the rules and regulations prior to the incident occurring although I admit it is a very complex problem with many ins and outs and a lot of what have you’s.

We have a number of other I.T Outsourcing related articles listed below that will provide you with more detailed information on a number of related topics:

https://optimizeddocs.com/blogs/consulting/consulting-index-page-01

Our team specializes in strategies for MSP Organizations and we assist in improving profit margins through standardization and consistent record keeping strategies, so you can be confident that our content is tailored to your needs.

Please feel free to explore our other articles and click on any that interest you. If you have any questions or would like to learn more about how we can help you with your documentation needs, please click the "Get In Touch" button to the left and we will be happy to assist you. Thank you for choosing us as your trusted source for technology documentation.